Library Imports

from pyspark.sql import SparkSession
from pyspark.sql import types as T

from pyspark.sql import functions as F

from datetime import datetime
from decimal import Decimal

Template

spark = (
    SparkSession.builder
    .master("local")
    .appName("Section 2 - Performing your First Transformations")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()
)

sc = spark.sparkContext

import os

data_path = "/data/pets.csv"
base_path = os.path.dirname(os.getcwd())
path = base_path + data_path

pets = spark.read.csv(path, header=True)
pets.toPandas()

Transformation

(
    pets
    .withColumn('birthday_date', F.col('birthday').cast('date'))
    .withColumn('owned_by', F.lit('me'))
    .withColumnRenamed('id', 'pet_id')
    .where(F.col('birthday_date') > datetime(2015,1,1))
).toPandas()

What Happened?

• We renamed the primary key of our df
• We truncated the precision of our date types.
• we filtered our dataset to a smaller subset.
• We created a new column describing who own these pets.

Summary

We performed a variety of spark transformations to transform our data, we will go through these transformations in detailed in the following section.